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Abstract. We use raising operators to prove a tableau formula for the eta 
polynomials of BKT3 and the Stanley symmetric functions which correspond 
to Grassmannian elements of the Weyl group W n of type D n . We define the 
skew elements of Wn and exhibit a bijection between the set of reduced words 
for any skew w £ W n and a set of certain standard typed tableaux on a skew 
shape X/fi associated to w. 



0. Introduction 

Let k be any positive integer and OG = OG(n — k, 2n) be the even orthogo- 
nal Grassmannian which parametrizes isotropic subspaces of dimension n — k in a 
complex vector space of dimension 2n, equipped with a nondegenerate symmetric 
bilinear form. Following [BKTlj . the Schubert classes u\ in H*(OG, Z) are indexed 
by typed k-strict partitions A. Recall that a fc-strict partition is an integer partition 
A = (Ai,...,A^) such that all parts Xi greater than k are distinct. A typed k- 
strict partition is a pair consisting of a fc-strict partition A together with an integer 
type(A) S {0, 1, 2} which is positive if and only if A, = k for some index i. 

In |BKT3j . we discovered a remarkable connection between the cohomology of 
even (type D) and odd (type B) orthogonal Grassmannians, which allows a uniform 
approach to the Schubert calculus on both varieties. We used this together with our 
earlier work |BKT2j to obtain a Giambelli formula for the Schubert class o\ , which 
expresses it as a polynomial in certain special Schubert classes. Moreover, we em- 
ployed Young's raising operators to define and study the eta polynomials H\{x ; y) 
- a family of polynomials which multiply like the Schubert classes on OG(rt — k, 2n) 
when n is sufficiently large. We proved that the polynomial H\(x;y) is equal to 
the Billey-Haiman Schubert polynomial 'D Wx (x ; y) indexed by the corresponding 
fc-Grassmannian element w\ in the Weyl group W n for the root system of type D„. 

The first goal of the present paper is to combine the above theory with the 
raising operator approach to tableau formulas developed in [T] . Our main theorem 
(Theorem [3]) is a formula for the eta polynomial H\(x ; y), which writes it as a sum 
of monomials (xy) u over certain fillings U of the Young diagram of A called typed 
k' -bitableaux. The analysis turns out to be rather subtle, since a direct combination 
of the results of [BKT3j and [T] does not lead to positive formulas. 

We call an element w of W n skew if there exists aw' € W n and a k- Grassmannian 
element w\ such that ww' — w\ and l(w) + l(w') — |A|. The skew elements of the 
symmetric group are precisely the 321-avoiding permutations or fully commutative 
elements, which were introduced in |BJS| and explored further in [Sll IS2j . In the 
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other classical Lie types, although every fully commutative element is skew, the 
converse is false. The skew elements of the the hyperoctahedral group were studied 
in [T], and we extend this theory here to W n . 

Let A and fj, typed fc-strict partitions such that the diagram of fi is contained 
in the diagram of A. Our approach leads naturally to the definition of certain 
symmetric functions Ex/^ix), given as a sum of monomials x T corresponding to 
typed k' -tableaux T on the skew shape A//z. We find that the function Ew^i), 
when non-zero, is equal to a type D Stanley symmetric function E w {x) indexed by 
a skew element w in W n . As in [BJS1 and [T]. there is an explicit bijection between 
the standard typed fc'-tableaux on the skew shape A/ fi and the reduced words for 
a corresponding skew element w G W n . 

This paper is organized as follows. Section [1] reviews the Giambelli and Pieri 
formulas which hold in the Chern subring of the stable cohomology ring of 
OG(n — k, 2n) as n — > oo. We also establish the mirror identity in , a key tech- 
nical tool which provides a bridge between the Pieri rule and our tableau formulas. 
Section [5] introduces the eta polynomials and proves various reduction formulas 
which arc then combined to obtain our main tableau formula for H\(x ; y). Finally, 
in Section |3] we relate this theory to the type D Schubert polynomials and Stanley 
symmetric functions of |BH1 [Lj , and study the skew elements of W n . 



1.1. Raising operators. An integer sequence is a sequence of integers a = {«i}i>i, 
only finitely many of which are non-zero. The largest integer £ > such that ai ^ 
is called the length of a, denoted £(a); we will identify an integer sequence of length 
I with the vector consisting of its first £ terms. We let \a\ ~^ai and #a equal the 
number of non-zero parts ai of a. We write a > f3 if a% > Pi for each i. An integer 
sequence a is a composition if ai > for all i and a partition if ai > ai+i > for 
all i. As is customary, we represent a partition A by its Young diagram of boxes, 
which has A^ boxes in row i for each i > 1. We write \x C A instead of /i < A for the 
containment relation between two Young diagrams; in this case the set-theoretic 
difference A \ /i is called a skew diagram and is denoted by A//i. 
Given any integer sequence a = a2, ■ ■ ■) and i < j, we define 



a raising operator R is any monomial in these Rij's. Given any formal power series 
J2i>o c iP in the variable t and an integer sequence a = (ai, 0% . . . , ai), we write 
Ca = c Ql c Q2 • ■ ■ c ai and set Rc a = CR a for any raising operator R. We will always 
work with power series with constant term 1, so that Cq = 1 and = for i < 0. 

1.2. The Giambelli formula. Fix an integer k > 0. We consider an infinite 
family wx,W2, ■ • ■ of commuting variables, with Wi of degree i for all i, and set 
u>o = 1, w r = for r < 0, and w a = w ai . Let 1^ C 1\w\, W2 : ■ ■ ■} be the ideal 
generated by the relations 



1. Preliminary results 



Rij( a ) = • • • > a i + 1) • • • i a j — 1) • • •); 



(1) 




for r > k. 



Define the graded ring — Z[w\, w 2 , . . 
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Let A° = £ NxN | 1 < i < j}, equipped with the partial order < defined 

by < (hj) h" an d only if i' < i and j' < j. A finite subset D of A° is a 

valid set of pairs if it is an order ideal, i.e., (i,j) € D implies £ D for all 

£ A° with < (i,j)- For any valid set of pairs D, we define the raising 

operator 

i<j i<j:(i,j)eD 

A partition A is k-strict if all its parts greater than fc are distinct. The k-length 
of A, denoted ^fc(A), is the cardinality of the set {i | Aj > k}. We say that A has 
positive type if Xi = k for some index i. The monomials wa for A a fc-strict partition 
form a Z-basis of C^. 

Given a fc-strict partition A, we define a valid set of pairs C(A) by 

(2) C(A) = {(*, j) € A | Aj + Aj > 2fe + j - i and j < £(A)} 

and set i? A := R c ^\ Furthermore, define W\ 6 by the Giambelli formula 

(3) •= w\. 
Since 

W^A = »A + ^ W M 

with the sum over fc-strict partitions fi which strictly dominate A, it follows that the 
W\ as A runs over fc-strict partitions form another Z-basis of . More generally, 
given any valid set of pairs D and an integer sequence a, we denote R D w a by W£. 

Example 1. In the ring we have 



W 32 2 = : ; „ (1 - -Ri3)(l - R23) W322 
I + H12 

= (I - 2R 12 + 2R\ 2 )(1 - R 13 )(l - R 23 ) w 322 

= (1 - 2R 12 + 2R\ 2 - 2R\ 2 )(1 — R 13 - R 23 + Ri 3 R 23 ) w 322 
= w 322 - W421 - 2w 7 + 2w 6 i - w 33 i + w i3 

— W 3 wl — — 2u)7 + 2wqWi — W 3 Wi + W4"W 3 . 

1.3. Cohomology of even orthogonal Grassmannians. Let the vector space 
V = C 2n+2 be equipped with a nondegenerate symmetric bilinear form. A subspace 
£ of V is called isotropic if the form vanishes when restricted to E. The dimensions 
of such isotropic subspaces E range from to n + 1 . 

Choose fc with < fc < n and let OG = OG(n + 1 — fc, 2n + 2) denote the Grass- 
mannian parametrizing isotropic subspaces of V of dimension n+l — k. Let Q denote 
the universal quotient vector bundle of rank n + 1 + fc over OG, and H*(OG,Z)i 
denote the subring of the cohomology ring H*(OG,Z) which is generated by the 
Chern classes of Q. There is a ring epimorphism ijj : — > H*(OG,Z)i which 
maps the generators w p to the Chern classes c p (Q) for each p > 1; in particular we 
have ip{ w p) = ii p > n + k. 

Let lZ(n, fc) denote a rectangle of size (n + 1 — fc) x (n + fc). It follows from 
BKT3[ Theorem 1] that for any fc-strict partition A, we have 



4 



HARRY TAMVAKIS 



fc 



s 



s 



Figure 1. Two fc'-related boxes in a fc-strict Young diagram 



if the diagram of A fits inside TZ(n,k), and ip(W\) — otherwise. Here Y\ is a 
certain Zariski closed subset of pure codimension |A| in OG, which is a Schubert 
variety in OG, if A does not have positive type, and a union of two Schubert varieties 
in OG, if A has positive type. The cohomology classes [Y\] for A a fc-strict partition 
contained in lZ(n,k) form a Z-basis for H*(OG,Z)i. More details and the proofs 
of these facts are provided in [BKT3] , 

1.4. The Pieri rule. We let [r, c] denote the box in row r and column c of a Young 
diagram. Suppose that c < fc < c'. We say that the boxes [r,c] and [r',c'] are fc'- 
related if c + c' = 2k + 1 + r — r' . In the diagram of Figure [TJ the two grey boxes 
are fc'-related. 

Given two partitions A and /i with AC/i, the skew Young diagram fi/X is called 
a horizontal strip (respectively, vertical strip) if it does not contain two boxes in 
the same column (respectively, row) . For any two fc-strict partitions A and fi, let a% 
(respectively Pi) denote the number of boxes of A (respectively fi) in column i, for 
1 < i < k. We have a relation A — > /i if /i can be obtained by removing a vertical 
strip from the first fc columns of A and adding a horizontal strip to the result, so 
that for each i with 1 < i < k, 

(1) if pi = on, then the box [c<i, i] is fc'-related to at most one box of /i \ A; and 

(2) if fa < ai, then the boxes [0i,i\,..., [ai,i] must each be fc'-related to exactly 
one box of fx \ A, and these boxes of fi \ A must all lie in the same row. 

If A — > fi, we let A be the set of boxes of /x x A in columns fc + 1 and higher 
which arc not mentioned in (1) or (2). Define the connected components of A by 
agreeing that two boxes in A are connected if they share at least a vertex. Then 
define -/V(A, /i) to be the number of connected components of A, and set 



We deduce from |BKT3[ Theorem 5] that the following Pieri rule holds: For any 
fc-strict partition A and integer p > 0, 




iV(A, fi) + 1 if A has positive type and /j, does not, 
N(\,u) otherwise. 




For any d > 1 define the raising operator by 



r *= n n a+^i)" 1 - 



l<i<j<d i<j : (i,j)eC(A) 
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We compute that 

l 

w p ■ W\ = w p ■ Re w\ = R( +1 ■ JJ(1 - Rii+iy 1 w\ iP 

i=l 

£ 

= Re+i ■ + ^+1 + R lw + • • • ) w ^p = E W " W ' 

where N = Af(X,p) is the set of all compositions v > A such that \v\ = \X\ + p and 
fj = for j > £ + 1, where £ denotes the length of A. Equation (Q| is therefore 
equivalent to the identity 

(5) J2 W » W = E 2 MM WW>, 
which was proved in [BKT3j . 

1.5. The mirror identity. Let A and v be fc-strict partitions such that 

vx > max(Ai,l(A) + 2k - 1) 

and p, m > 0. Then the Pieri rule ((H) implies that the coefficient of W v in the 
Pieri product w p ■ W\ is equal to the coefficient of W(ui+m,vi,va,...) in the product 
Wp+m ■ W\ . We apply this to make the following important definition. 

Definition 1. Let A and p be fc-strict partitions with /j, C A, and choose any 
p > max(Ai+l,^(A)+2fc— 1). If |A| = |/x|+r and A — > (p+r,n), then we write /i A 
and say that A//i is a k' -horizontal strip. We define m(A//i) :— M(X, (p + r, fi)); 
in other words, the numbers m(X/ fi) are the exponents that appear in the Pieri 
product 

(6) Wp.W x =Y,2 m[X/,l) W ( p +r ^ 

with the sum over integers r > and fc-strict partitions /i C A with = |A| — r. 

Note that a fc'-horizontal strip A//x is a pair of partitions A and \x with fi X. 
As such it depends on A and fi and not only on the difference A \ /i. 

£+1 1 - R 

Lemma 1. Let ^ = J^J ^ — , and suppose that we have an equation 



E 



m C^, where the sums are over all v — (v\, . . . , i/f), while a v and b v are integers 
only finitely many of which are non-zero. Then we have 



w 



in the ring C {k \ for any integer p. 

Proof. The proof is the same as [U Proposition 2]. □ 
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Let A be any fc-strict partition of length I. Consider the following version of (0 : 

(7) £ WfgJ = £ 2^-> W c ^ 

a>0 

where the first sum is over all compositions a of length at most I + 1, and the 
second over fc-strict partitions /i with A — > /i. Following [H §4], the next result is 
the mirror identity of ([7]) . 

Theorem 1. For A any k-strict partition we have 

(8) 2#Q W \-l = E 2 m{x '^ W„, 

a>0 

where the first sum is over all compositions a. 

Proof. Choose p > |A| + 2k and let I = ^(A). Expanding the Giambelli formula 
with respect to the first row gives 

e+1 l + Rl 



'p ■ W\ — r c ^ [] «»o,a) = ^ C(P ' A) IK 1 + 2R ^ + 2R h + • • • ) »wo 

= Y2* a W? { i\ x \, v 

' ' (p+\a\,X—a) 



a>0 

Comparing this with (j6|), we deduce that 

V*) VV (p+\a\,\-a) — "(P+|A|-|/x|,/i)- 

We claim that for every integer r > 0, 

(io) X 2# XK- a) = E tf^ww)- 

1 |Ml=|A|-r 

The proof is by induction on r. The base case r = is clearly true. For the 
induction step, suppose that we have for some r > that 



(id EE 2#a «L, = E E 2 " i(A/ ^< 



(p+S,/i)- 



s>r |q|— s s>r 

~ ~ mI = |a|- 



Expanding the Giambelli formula with respect to the first component, we obtain 
Wp+ S W^_^ as the leading term of W^^'^ x _ a y while w p + s is the leading term of 
Wfp+ St) A. Since the set of all products Wd W v for which (d, v) is a fc-strict partition 
is linearly independent in , we deduce from dTTT) that 

(12) £2 #a wfl A j= Y 2 m ^/^W„. 

\a\=r f'" A 

By applying Lemma [T] we see that (|10[) is true, and this completes the induction. 
This also finishes the proof of Theorem [TJ since the argument shows that (TT2"]) holds 
for every integer r > 0. □ 
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Figure 2. Two A:"-related boxes in a Young diagram 

1.6. We say that boxes [r, c] and [r', c 1 ] are k" -related if |c — k\ + r = \d — k\ + r 1 
(we think of k" as being equal to k — 1). For example, the two grey boxes in the 
diagram of Figure [2] are fc"-related. We call box [r, c] a left box if c < k and a right 
box if c > k. 

If /x C A are two fc-strict partitions such that A//i is a ^'-horizontal strip, we 
define \q = [1q = oo and agree that the diagrams of A and fj, include all boxes 
[0, c] in row zero. We let R (respectively A) denote the set of right boxes of // 
(including boxes in row zero) which are bottom boxes of A in their column and are 
(respectively are not) A;"-related to a left box of A//u. Let N(A) denote the number 
of connected components of A. Moreover, define 

n(A//i)=4(M)-4(A)+m(A//x). 

Lemma 2. A pair fi C A of k-strict partitions forms a k' -horizontal strip A/// if 
and only if (i) A//i is contained in the rim of X, and the right boxes of X/fj, form a 
horizontal strip; (ii) no two boxes in R are k" -related; and (Hi) if two boxes of Xf [i 
lie in the same column, then they are k" -related to exactly two boxes of R, which 
both lie in the same row. We have 

(13) "(A/ ) i ^ ^ ^ as P os ^ ve tyP e an d H does not, 

1 AT (A) — 1 otherwise. 

Proof. We have (J, A if and only if A — > (p + r, fi) for any p > \X\ + 2fc, where 
r = | A — fi\. Observe that a box of (p + r, fi) \ A corresponds to a box of fi which is a 
bottom box of A in its column. The rest of the proof is a straightforward translation 
of the definitions in ijl.41 □ 



2. Reduction formulas and tableaux 

2.1. Let x = (x\,x-2,..^) and set y — (yi,---,Uk) for a fixed integer k > 1. 
Consider the generating series 

oo i , oo k oo 

IiT^-E^K" ^d H(l + yj t)=Y,er(y)t r 

i=l 1 r=0 j=l r=0 

for the Schur Q-functions q r (x) and elementary symmetric functions e r (y). Given 
any strict partition A of length £(X), the Schur Q-function Q\(x) is defined by the 
raising operator expression 



qa = n i " ^ 



i<j 



R, qx 



■ij 
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and the P- function is given by P\{x) — 2 e W Q\(x). In particular Pq = 1 and for 
each integer r > 1, we have P r = q r /2. For each integer r, define $ r = •d r (x ; y) by 

#r = },Qr-i(x)ei(y). 

i>0 

We let r( fe ) = Z[$i,$2,$3, ■ • ■] be the ring of theta polynomials. There is a ring 
isomorphism C^ k > — > r( fe ) sending w r to $ r for all r. 

Let A be a fc-strict partition, and consider the raising operator 

i<j i<j : A i +A i >2fe+j-j 

The theta polynomial 8a (x ; y) is defined by the equation ®\ = R x i3 x . The poly- 
nomials 0a for all fc-strict partitions A form a Z-basis of T^ k \ 
Set 



Vr{x;y) 
lfc-1 



er(y) +2^i=o • p r-t(a:)et(y) if r < fc, 



> Z)i=o- P r-i(a:)e<(y) ifr>fc 
and ^(a; ; y) = Yh=o Pk-i(x)ei(y) . For any r > 0, we have 

t] r if r < fc, 

(14) tf r = < r? fc + ^ if r = fc, 

2^,, if r > fc, 

while i]k — rj' k = ek(y). Following BKT.'» , we define the ring of eta polynomials 

B {k) = Z[j7i,. . .,rj k -i,r]k>ilk' r lk+i ■ ■ •]• 
2.2. For any fc-strict partition A, define Q x (x ; y) and H x (x ; y) by the equations 

9 A = R X i?a and H x = 2- £kW 6 A . 
Proposition 1. For any k-strict partition X, we have the reduction formula 



p—0 ti~*x 

M = |A|-P 



(15) H x (x;y)=J2x p 1 ]T 2*W»> H M {x;y). 
Proof. We compute that 

oo 1 oo k 

£ ; v)r - ^ n y^i IK 1 + w*) - E *t 2# * £ ; 

r=0 1 i=2 1 j=l i=0 s=Q 

and therefore, for any integer sequence A, we have 

(16) # x (x;y) = J2xi l2#a #X-<*(Z\y) 

a>0 

summed over all compositions a. If R denotes any raising operator, then 
Rti x (x ; y) = $ RX (x ;y) = £ 4°' 2 *° *nx-«(x ; y) = £ 4°' 2* a R$x- a (Z ; y). 

Applying the raising operator R x to both sides of (fl~6|) . we therefore obtain 

oo 

q x (x ; y) = £ 4 Q| 2 #Q §^(i ;y) = £*? £ 2#Q § A^(i ; v), 

Q>0 p=0 |«|=P 
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where &x_a = R x $\- a by definition. We now use the mirror identity (j8|) and the 
equation 6a = 2 (k ^ H\ to complete the proof. □ 

For any fc-strict partition A of positive type, we define H\{x ; y) by the equation 

(17) H x = 2- e *We k (y)O x _ k 

where A — fc means A with one part equal to k removed. If A does not have a part 
equal to k, we agree that H\ = 0. The following analogue of Proposition [1] is valid 
for the polynomials H(x ; y). 

Proposition 2. For any k-strict partition A of positive type, we have the reduction 
formula 

oo 

(18) H x (x;y) = J2 x i E F Wlt) H^iv) 

M=lM-» 

where the sum is over k-strict partitions \x C A of positive type such that A//i is a 
k' -horizontal strip. 

Proof. This follows immediately from (|17l) and the reduction formula for the theta 
polynomial ®\-k(x ;y). For the latter, see [TJ Theorem 4], □ 

2.3. Typed fc-strict partitions and eta polynomials. A typed k-strict partition 
A consists of a fc-strict partition (Ai,...,A^) together with an integer type(A) G 
{0, 1,2}, such that type(A) > if and only if Xj = k for some index j. Suppose 
that A is a typed fc-strict partition and R is any finite monomial in the operators 
Rij which appears in the expansion of the power series R x in ([3]). If type(A) = 0, 
then set R-kfl^ = $#a- Suppose that type(A) > 0, let m = ^(A) + 1 be the 
index such that A m = k < A m _i, and set a — (or, . . . , a m _i, a m +i, ■ ■ ■ , cue) for any 
integer sequence a of length I. If R involves any factors Rij with i — m or j = m, 
then let R-k'&x — \ "&r.\. If R has no such factors, then let 



jfk^Rx ifty P e(A) = 2. 



The eta polynomial H\ = H\(x;y) is the element of defined by the raising 
operator formula 

(19) H x = 2- ekW R x *i9 X - 

The type of the polynomial H\ is the same as the type of A. It was shown in |BKT3| 
Thm. 4] that the H\ for typed fc-strict partitions A form a Z-basis of the ring B^ k \ 
Recall that OG = OG(rt + l — fc, 2n + 2) denotes the Grassmannian parametrizing 
subspaces of C 2n+2 of dimension n + 1 — fc which are isotropic with respect to an 
orthogonal bilinear form. There is a Schubert class o\ in H*(OG, Z) for every typed 
fc-strict partition A whose diagram fits inside the rectangle lZ(n,k). Moreover, we 
have a ring epimorphism — > H*(OG, Z) which maps H\ to o~\, whenever the 
diagram of A fits in lZ(n, fc), and to zero, otherwise. 
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For any typed fc-strict partition A, we define H\ and H\ as above, by ignoring 
the type of A. It follows that the definition (fT9]) of H\ is equivalent to the formula 



[ H X if type(A) = 0, 

(20) H x = <{Hx + Hx)/2 if type(A) = 1, 

[ (Hx - H x )/2 if type(A) = 2. 

Lemma 3. Let X be a typed k-strict partition and Hx(0;y) be obtained from 
Hx(x ; y) by substituting xi — for all i > 1. Then we have 



Hx(0;y) 



'0 if Ai > k, 

sy{y) ifX\=k and type(A) = 1, 

if Ai = k and type(A) = 2, 

.SA'(y) if\i<k 

where A' denotes the partition conjugate to A. 
Proof. The raising operator definition of A gives 

(21) Q x (0;y) = R x e x (y) 

where ex = Yii iv) an d e r (y) denotes the r-th elementary symmetric polynomial 
in y. Since e r (y) = for r > k, we deduce from (|21l) that Hx(0;y) = unless 
Ai < k. In the latter case we have £k(X) = 0, C(X) = 0, and 

(22) 6 A (0 ; y) = 6 A (0 ; y) = - R l0 ) e x {y) = s y (y). 

i<j 

Suppose that Ai = k, so that type(A) > 0. If typc(A) = 1, then (J2DJ and give 



^a(0 \V) = \ (e A (0 ; y) + e fe (y)e A - fe (0 ; y)) 



= 2 ( s A'(y) + efc(y)s( A -fc)'(0;y)) = sv(y) 5 

since ek(y) s (\—k)'(y) = s \'(y) by the classical type A Pieri rule. We similarly 
compute that Hx(0 ;y) = if type(A) = 2. Finally, if Ai < /c then type(A) = and 
H x (0 ; y) = 6 A (0 ; y ) = s x > (y) , using © again. □ 

2.4. Tableau formulas. In this subsection we will obtain a tableau description 
of the eta polynomials Hx(x ; y), where the tableaux in question are fillings of the 
Young diagram of A. We first prove a reduction formula for the x variables in Hx- 

Definition 2. If A and [i are typed fc-strict partitions with /i C A, we write [i A 
and say that X//J, is a typed k' -horizontal strip if the underlying fc-strict partitions 
are such that A/ ' \x is a fc'-horizontal strip and in addition type(A) + type(/i) ^ 3. In 
this case we set n(X/fi) — N(A) — 1, where the set A is defined as in 31.61 



Theorem 2. For any typed k-strict partition X, we have the reduction formula 



(23) Hx(x;y) = Y J *l £ H„{x;y) 

lxl=|X|-p 

where the inner sum is over typed k-strict partitions [i with fi ~~> A and = |A| —p. 
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Proof. Assume that type(A) = 1; the proof in the other cases is similar. Using 
Definition IT9l and equations (fT5|) and (fT8|) we obtain 

oo 

(24) H x (x;y) = J24 £ 2*<Vm) - y ) + H„(x ;y))/2 

n=0 A 

l»| = |X|-p 

where the inner sum is over all fc-strict partitions /icA with \fi\ = \X\—p such that 
X/li is a fc'-horizontal strip. The result follows by combining equation (|24l) with 
(fTB) and ||3DJ). □ 

Let P denote the ordered alphabet {1' < 2' < ■ ■ • < fc' < 1,1° < 2, 2° < ■ ■ • }. 
The symbols 1', . . . , k' are said to be marked, while the rest are unmarked. Suppose 
that A is any typed /c-strict partition. 

Definition 3. a) A typed k' -tableau T of shape X//J, is a sequence of typed /c-strict 
partitions 

(25) fi = X° C A 1 C • • • C A r = A 

such that X l /X l ~ 1 is a typed fc'-horizontal strip for 1 < i < r. We represent T by a 
filling of the boxes in X/ /j. with unmarked elements of P which is weakly increasing 
along each row and down each column, such that for each i, the boxes in T with 
entry i or i° form the skew diagram A l /A* _1 , and we use i (resp. i°) if and only if 
type(A l ) ^ 2 (resp. type(A*) = 2), for every i > 1. A standard typed k! -tableau on 
Xj\x is a fc'-tableau T of shape A//x such that the entries 1, 2, . . . , |A — fi\, circled or 
not, each appear exactly once in T. For any typed fc'-tableau T we define 

(26) n (T) — £ n(A i /A i_1 ) and x T =^ X T 

i i 

where m, denotes the number of times that i or i° appears in T. 

b) A typed k! -bitableau U of shape A is a filling of the boxes in A with elements of 
P which is weakly increasing along each row and down each column, such that (i) 
the unmarked entries form a typed fc'-tableau T of shape A//x with type(/i) ^ 2, 
and (ii) the marked entries are a filling of \i which is strictly increasing along each 
row. We define 

k 

n(U) = n(T) and (xy) u = x T J| yj 3 

j=i 

where nj denotes the number of times that j' appears in U. 
Theorem 3. For any typed k-strict partition X, we have 



(27) H x (x;y)=Y,2 n{U) (xy) 1 



u 

where the sum is over all typed k! -bitableaux U of shape X. 



Proof. Let m be a positive integer, x^ — {x%, . . . , x m ), and let Hx(x^ ; y) be the 
result of substituting Xi = for i > m in H\(x;y). It follows from equation (|23p 
that 

oo 

(28) H x (x^;y)=J2< £ H,(x^ ;y). 
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Iterating equation ([28]) m times produces 

where the sum is over all typed A:-strict partitions /i c A and fc-tableau T of shape 
X/fi with no entries greater than m. We deduce from Lemma [3] that H^{Q]y) = 
unless fix < k and type(/i) ^ 2, in which case H^(0 ; y) = s^> (y). The combinatorial 
definition of Schur S- functions [M] 1.(5.12)] states that 

(29) v(y) = E?/ 5 

s 

summed over all semistandard Young tableaux S of shape /x' with entries from 1 
to k. We conclude that 

H x (x^;y) = J2 2" (C/) <FU) V 
u 

summed over all typed fc'-bitableaux U of shape A with no entries greater than m. 
The theorem follows by letting m tend to infinity. □ 

Example 2. Let k = 1, A = (3, 1) with type(A) = 1, and consider the alphabet 

Pi, 2 = {1' < 1, 1° < 2, 2°}. There are twelve 1-bitableaux U of shape A with entries 

1' 1 2 1' 2 2 1' 1 2 

in Pi, 2- The bitableaux , , ^ , and ^ satisfy n(U) = 1, while the 

ten bitableaux 

111 112 122 122 I'll 
2 2 12 1 

I'll 1'12 1'2 2 I'll 1/2 2 
2 2 2 1' 1' 

satisfy n(U) — 0. We deduce from Theorem |3] that 

#3,1(21, x 2 ;yi) = {x\x 2 + 2x\x\ + x\x\) + [xf + Zx\x 2 + Zxyx\ + x\) y\ 

+ (x 1 + 2x\x 2 + x 2 ) y\ 

= #3,1(^1, x 2 ) + (#3(21,22) + #2,1(21, x 2 )) yi + P 2 (xi 1 x 2 )yj. 

We similarly find that for A = (3, 1) of type 2, there are six 1-bitableaux U of shape 
A with entries in Pi,2, namely 

111 112° 1°2°2° 1°2°2° I'll 1'12° 
2° 2° 1° 2° 2° 2° 

all of which satisfy n(U) = 0. We deduce from Theorem [3] that 

#3,1(21,22 ; yi) = {x\x 2 + 2x\x\ + x\x\) + (xfx 2 + X\X%) y\ 

= #5,1(21,22) + #2,1(21, £2)3/1- 

Notice that #3,1 + H' z l = 5O3.1, and compare with (TJ Example 7]. 

Theorem [3] motivates the following definition. 

Definition 4. For A and fi any two typed fc-strict partitions with /x C A, let 

E x/ ,(x) =J2^ T) ^ T 

T 
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where the sum is over all typed fc'-tableaux T of shape A/ ' ji. 

Example 3. (a) There is an involution j on the set of all typed fc-strict partitions, 
which is the identity on partitions of type and exchanges type 1 and type 2 
partitions of the same shape. If T is a typed fc'-tableau of shape A//z then we let 
j(T) be the typed fc'-tableau of shape j(X)/j([i) obtained by applying j to each 
typed partition X 1 which appears in the sequence (|25j) determined by T. Then j 
is an involution on the set of all typed fc'-tableaux and it follows from Definition 0] 
that E x /p{x) = E jW /j( M )(x). 

(b) Suppose that m > min(fc,Ai) for all i. Then Definition [4] becomes the tableau 
based definition of skew Schur P-functions, and therefore Ex/^x) = Px/^{x). 

(c) Suppose that Ai < k, so in particular C(A) = 0. Then n(A//x) is equal to the 
number of edge-connected components of A//i, for any partition /icA. It follows 
from Worley [SO §2.7] that 

J2 2" (T) x T = Sx/^x) := det^-^+j-^a:))^-- 

T 

We therefore have Ex/^{x) — Sx/^{x) and 

H\{xyy) = Sx/,i(x) s^(y), 

/jCA 

in agreement with [BKT3, Prop. 5.4(a)]. 

(d) Suppose that there is only one variable x. Then we have 
' 2 «(a/m) x |A-mI if A//i is a fc'-horizontal strip, 



otherwise. 



Corollary 1. (a) Let x — (xi, X2, ■ ■ ■) and x' = (x[, x' 2 , . . .) be two sets of variables, 
and let A be any typed k-strict partition. Then we have 

(30) H x (x, x' ; y) = Ex/n (x) H, t (x' ; y) , 

MCA 

(31) Hx{x;y) = Y,E x/ll {x)s tl ,{y) t 
and 

(32) E x (x,x') = J2 E x/^)M^')^ 

where the sums are over all typed k-strict partitions /icA. 

(b) For any two typed k-strict partitions X and fi with fi C A, we have 

(33) Ex/n(x,x') = ^2e x / u (x) E u/ll (x'), 

where the sum is over all typed k-strict partitions v with fi C v C A. 

Proof. The proof is the same as that for Corollaries 6 and 7 in [T] . □ 

It follows immediately from identity ()30|) that E x /^(x) is a symmetric function 
in the variables x. These functions will be studied further in the next section. 
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3. Stanley symmetric functions and skew elements 

3.1. Let W n +i be the Weyl group for the root system of type D n +i, and set 
Woo = Un Wn+l' The elements of W n +i may be represented as signed permutations 
of the set {1, . . . , n+1}; we will denote a sign change by a bar over the corresponding 
entry. The group W n+ i is generated by the simple transpositions Sj = (i, i + 1) for 
1 < % < n, and an element Sq which acts on the right by 

(Ul, «2) • • -j "n+l)so = (u 2 ,Ul,U3, . . . ,U n+1 ). 

Every element w G Woo can be expressed as a product of w = s ai s a2 ■ ■ ■ s ar of simple 
reflections s j . If the length r of such an expression is minimal then the sequence of 
indices (ai, . . . , a r ) is called a reduced word for w, and r is called the length of u>, 
denoted A factorization w — uv in Woo is reduced if £(iu) = ^(w) + ^(^)- 

Following |F51lFETllFE2l lL]. we will use the nilCoxeter algebra W n+1 of W„+i 
to define type D Stanley symmetric functions and Schubert polynomials. W n +i 
is the free associative algebra with unity generated by the elements uq, Ui, . . . , u n 
modulo the relations 

uf = i > ; 

u u 2 u a = U 2 U U 2 
UiU i+1 Ui = u i+1 UiU i+ i i > ; 

UiUj — UjUi j > i + 1, and ^ (0,2). 

For any w e W n +i, choose a reduced word a\---ai for w and define u m = 
u ai . . . u ae . Since the last four relations listed are the Coxeter relations for D n+ i, it 
is clear that u w is well defined, and that the u w for w G W n +i form a free Z-basis 
of W n+ i. 

Let t be an indeterminate and define 

Ai(t) = (1 + iu„)(l + tu n -i) ■■■{! + tui) ; 
D(t) = (1 + tu„) •••(!+ tu 2 )(l + + <tt )(l + <u 2 ) • • • (1 + 

According to [L] Lemma 4.24] , for any commuting variables s, £ we have D(s)D(t) = 
D(t)D(s). Set x = (x±,x 2 , ■ ■ ■) and consider the product D(x) := D(xi)D(x 2 ) 
We deduce that the functions E w (x) in the formal power series expansion 

(34) D{x) = J2 E w {x)u w 

are symmetric functions in x. The E w are the type D Stanley symmetric functions, 
introduced and studied in |BH1 [L] . 

Let y = (yi, y 2l . . .). The Billey-Haiman type D Schubert polynomials T) w (x ; y) 
for w £ W n +i arc defined by expanding the formal product 

(35) D{x)A 1 {y 1 )A 2 (y 2 )---A n {y n ) = ^ D w (x;y)u w . 

The above definition is equivalent to the one in |BH] . One checks that 53 w is stable 
under the natural inclusion of W n in W n +i, and hence well defined for w G W^. 
We also deduce the following result from (jM)) and (1331) . 



A TABLEAU FORMULA FOR ETA POLYNOMIALS 



15 



Proposition 3. Let w e Woo and x' — (x' 1 ,x' 2 , . . .). Then we have 

(36) E w {x,x')= E u {x)E v {x') 

uv—w 

and 

(37) D w {x,x';y) = ^ E u {x)D v {x' ;y) 

uv—w 

where the sums are over all reduced factorizations uv — w in Woo ■ 

3.2. For k 1, an element w £ Woo is fc-Grassmannian if £(wsi) = £(w) + 1 
for all i ^ k. We say that w is 1-Grassmannian if £(wsi) — £(w) + 1 for all 
i > 2. The elements of W n +i index the Schubert classes in the cohomology ring 
of the flag variety S02 n +2 /B, which contains H*(OG(n + 1 — k,2n + 2),Z) as 
the subring spanned by Schubert classes given by fc-Grassmannian elements. If 
V(k, n) denotes the set of typed fc-strict partitions whose Young diagrams fit inside 
an (n + 1 — k) x (n + k) rectangle, then each typed fc-strict partition A in V(k,n) 
corresponds to a fc-Grassmannian element u>\ € W n +i which we proceed to describe. 

The typed fc-strict partitions A in V(k, n) are those whose Young diagram fits 
inside the non-convex polygon II obtained by attaching an (n + 1 — fc) x fc rectangle 
to the left side of a staircase partition with n rows. When n = 7 and fc = 3, the 
polygon II looks as follows. 



n 



The boxes of the staircase partition that are outside A lie in south-west to north- 
east diagonals. Such a diagonal is called related if it is fc'-related to one of the 
bottom boxes in the first fc columns of A, or to any box [0,i] for which Ai < i < fc; 
the remaining diagonals are called non-related. Let u\ < < • • • < Uk denote the 
lengths of the related diagonals. Moreover, let A 1 be the strict partition obtained 
by removing the first fc columns of A, let A 2 be the partition of boxes contained in 
the first fc columns of A, and set r = ^(A 1 ) = ^(A). 

fc 



If type(A) = 1, then the fc-Grassmannian element corresponding to A is given by 



W\ = (Ut + 1, . . . , U k + 1, (A 1 )! + 1, . . . , (A X ) r + 1, 1, Vi + 1, . . . , V n -k-r + 1), 

while if type(A) = 2, then 



w\ = (ui + 1, . . . ,u k + 1, (A : )i + 1, . . . , (A 1 ),. + 1, l,«i + 1, . . . ,w„_ fe _ r + 1), 
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where v\ < ■ ■ ■ < v n -k-r are the lengths of the non-related diagonals. Here 1 = 1 or 
1, so that the total number of barred entries in w\ is even. Finally, if type(A) = 0, 
then u\ = 0, that is, one of the related diagonals has length zero. In this case 

W X = (1,1*2 + 1, ■ • ■ + 1, (A 1 )! + 1, . . . , (A 1 ),. + l,Vi + 1, . . . ,V n+ i- k -r + 1), 

where v\ < ■ ■ ■ < Un+i-fc-r are the lengths of the non-related diagonals. For 
example, the element A = (8,4,3,2) G 7 3 (3,7) of type 1 corresponds to w\ = 
(3,5,7,6,2,1,4,8). 



A = 
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o 
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o 


o 
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o 













The element w\ € Woe depends on A and fc, but is independent of n. Further- 
more, it was proved in |BKT3| Prop. 6.3] that 

(38) H x {x-y)=® Wx (x-y) 
for any typed fc-strict partition A. 

3.3. The following definition can be formulated for any Coxeter group. 

Definition 5. An element w S Woo is called skew if there exists a typed fc-strict 
partition A (for some fc) and a reduced factorization w\ = ww' in Woo- 

Note that if we have a reduced factorization w\ — ww' in W n +i for some typed 
partition A G P(k,n), then the right factor w' is fc-Grassmannian, and therefore 
equal to w^ for some typed fc-strict partition /i 6 V(k 1 n). 

Proposition 4. Suppose that w is a skew element of Woo, and let A and fj, be typed 
k-strict partitions such that the factorization w\ — ww^ is reduced. Then we have 
fi C A and E w (x) = Ex/^x). 

Proof. By combining (|38|) with (|30[) and ([37| . we see that 

(39) Y, E V^ x ) H ^ x '>y) = H ^ x i x '>y)= E E u ( x )D v ( X ';y) 

(iCA uv=w x 

where the first sum is over all typed fc-strict partitions /i C A and the second sum 
is over all reduced factorizations uv — w\. The right factor v in any such reduced 
factorization of w\ must equal w v for some typed fc-strict partition r/, and therefore 
S)„(a;' ;y) = H„(x' ;y). Since the H u (x' ;y) for v a typed fc-strict partition form a 
Z-basis for the ring (x' ; y) of eta polynomials in x' and y, the desired result 
follows. □ 

Definition 6. Let A and [i be typed fc-strict partitions in V(k, n) with /i C A. We 
say that (A,/Lt) is a compatible pair if there is a reduced word for w\ whose last \/i\ 
entries form a reduced word for w^\ equivalently, if we have Ifaxw^ 1 ) = |A — fj,\. 

From Proposition 3] we immediately deduce the next result. 
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Corollary 2. Let (A,/i) be a compatible pair of typed k-strict partitions. Then 
there is a 1-1 correspondence between reduced factorizations of w\w~ l and typed 
k-strict partitions v with /i C v C A such that (A, v) and (u, /x) are compatible pairs. 

Theorem 4. Let A and (i be typed k-strict partitions in V(k,n) with /i C A. Then 
the following conditions are equivalent: (a) Bw^fi) ^ 0; (b) (A,/i) is a compatible 
pair; (c) i/tere exists a standard typed k' -tableau on X/fi. If any of these conditions 
holds, then E x/li {x) = E WxW -i(x). 

Proof. Equation (|39[) may be rewritten in the form 

(40) J2 E v» (*) H v ( x> ; v) = E ^x-,- 1 ( s ) ^ ; w) 

MCA |U 

where the second sum is over all (J, C A such that (A, /i) is a compatible pair. It 
follows that 

. , \E.„,-i(x) if (A, u) is a compatible pair, 

10 otherwise 

and we deduce that (a) and (b) are equivalent. Suppose now that (A,/Lt) is a 
compatible pair with |A| = \fi\ + 1, so that w x — s^w^ for some i > 0. Observe 
that if a; is a single variable, then E Si (x) = x, if i < 1, and E Si (x) = 2x, if i > 1, 
and therefore Ex/^ix) ^ 0. We deduce from Example 0(d) that A//i must be a 
^'-horizontal strip. Using Corollary^ it follows that there is a 1-1 correspondence 
between reduced words for w\w~ l and sequences of typed fe-strict partitions 

At = A° C A 1 C ■ ■ • C A r = A 

such that |A l | = |A i_1 | + l and A l /A i_1 is a fc'-horizontal strip for 1 < i < r = |A— fi\. 
The latter objects are exactly the standard typed fc'-tableaux on X/fx. This shows 
that (b) implies (c), and the converse is also clear. □ 



The previous results show that the non-zero terms in equations (|30[) and 
correspond exactly to the terms in equations ((37)) and (|36|) , respectively, when 
w = w\ is a fc-Grassmannian clement of Woo. 

Corollary 3. Let w £ Woo be a skew element and (A, fi) be a compatible pair such 
that w\ = wWfj, . Then the number of reduced words for w is equal to the number of 
standard typed k' -tableaux on X/fi. 

Let i denote the involution on the set of reduced words which interchanges the 
letters and 1, for example t(02120) = 12021. Then the restriction of i to the set 
of reduced words representing skew elements of Woo corresponds to the involution 
j on the set of typed /c'-tableaux defined in Example [^a). 

Example 4. Let A be a typed fc-strict partition with type(A) E {0, 1} and let A 1 
and A 2 be defined as in H3.2I We can form a standard typed fc'-tableau on A by 
filling the boxes of A 2 , going down the columns from left to right, and then filling 
the boxes of A 1 , going across the rows from top to bottom. When k = 2 and 
A = (7, 6, 5, 2) with type(A) = 1, the typed 2'-tableau on A which results is 

1 5 9 10 11 12 13 

2 6 14 15 16 17 

3 7 18 19 20 

4 8 
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which corresponds to the reduced word 

32043215432043215432 

for the 2-Grassmannian element w\ = 236541 6 Dq. We can form a reduced word 
for the element w' x = 236541 associated A = (7, 6, 5, 2) with type(A) = 2 by applying 
the involution t to obtain 

32143205432143205432. 
This last word corresponds to the standard typed 2'-tableau 

1 5° 9° 10° 11° 12° 13° 

2 6° 14° 15° 16° 17° 

3 7° 18° 19° 20° 

4 8° 

Example 5. Consider the 1-Grassmannian element w = 3421 in Wi associated to 
the typed 1-strict partition A = (4, 2, 1) of type 1. The following table lists the nine 
reduced words for w and the corresponding standard typed 1-tableaux of shape A. 



word tableau 


word tableau 


word tableau 


14 5 6 
1320321 2 7 
3 


13 5 6 
1323021 2 7 
4 


13 4 5 
1232021 2 7 
6 


12 3 5 
1230201 4 7 
6 


1° 2 3 5 
1230210 4 7 
6 


12 3 4 
1203201 5 7 
6 


1° 2 3 4 
1203210 5 7 
6 


14 5 7 
3120321 2 6 
3 


13 5 7 
3123021 2 6 
4 



Corollary 4. For any two typed k-strict partitions A, /i with /i C A, the function 
Ex/nix) is a nonnegative integer linear combination of Schur P -functions. 

Proof. According to |BH] and [L] , the type D Stanley symmetric function E w (x) is 
a nonnegative integer linear combination of Schur P-functions. The result follows 
from this fact together with Theorem |U □ 
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